Using environment and user data to deliver advertisements targeted to user interests, e.g. based on a single command

ABSTRACT

A system and an associated method for responding to a user&#39;s voice inquiry are disclosed. The system accepts the voice inquiry and obtains personal data regarding the user. The system then identifies potential subjects of interest in the voice inquiry from media content currently provided to the user through a device which has captured the voice inquiry, media content present in or capturing the user&#39;s surroundings, or media content previously provided to the user as responses to previous voice inquiries by the user. Next, the system determines at least one subject of interest based on at least one of the personal data and the user&#39;s previous voice inquiries. The system then presents a response related to the determined subject of interest to the user&#39;s voice inquiry.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a division of, and claims the benefit of, U.S.patent application Ser. No. 14/447,414, entitled “USING ENVIRONMENT ANDUSER DATA TO DELIVER ADVERTISEMENTS TARGETED TO USER INTERESTS, E.G.BASED ON A SINGLE COMMAND,” filed Jul. 30, 2014, which is a continuationof, and claims the benefit of, U.S. patent application Ser. No.14/316,663, entitled “VOICE-CONTROLLED INFORMATION EXCHANGE PLATFORM,SUCH AS FOR PROVIDING INFORMATION TO SUPPLEMENT ADVERTISING,” filed Jun.26, 2014 (now U.S. Pat. No. 9,639,854, issued May 2, 2017, whichapplications are incorporated herein by reference in their entirety.

BACKGROUND

An advertisement is effective when an audience heed's its message. Theaudience may be encouraged to take action or consider an idea. And ifthe message is persuasive enough, the audience acts as the advertiserdesires it to.

Much of the effectiveness of an advertisement lies in its ability toconvey information. On radio and television, for example, advertisementsfor products are played during breaks in programming or included in theprogramming itself, such as by product placement. And online,advertisements are displayed adjacent to a web page's content or as partof it. These advertisements may convey information by telling a story oroffering an opinion or a fact. Through the presentation of its message,advertisers hope to communicate the right information to persuade itsaudiences to follow a desired course of action.

But often an advertisement is unpersuasive. One common reason for thisis that the audience did not learn information relevant to itsinterests. Some types of advertisements provide avenues for an audienceto learn more about a topic of choice. For example, a person activelyconsuming media content by browsing web pages can select a hyperlinkdisplayed on a website to learn more information about an advertisedtopic. But when media is consumed more passively, such as throughtelevision, radio or any device in the environment, a person might noteven know the identity of an item of interest and would typically needto search for more information regarding the item of interestautonomously and strenuously. Doing so is generally cumbersome andinefficient, thus reducing the likelihood that the person receivesappropriate informational advertisement and is ultimately persuaded bythe advertisement. As a result, many advertisers lose opportunities topersuade audiences that would likely be receptive toward their messages.

The need exists for a system that overcomes the above problems, as wellas one that provides additional benefits. Overall, the examples hereinof some prior or related systems and their associated limitations areintended to be illustrative and not exclusive. Other limitations ofexisting or prior systems will become apparent to those of skill in theart upon reading the following Detailed Description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a suitable environment in which avoice-controlled advertisement platform operates.

FIG. 2 is a diagram illustrating an example infrastructure for utilizinga software development kit for the voice-controlled advertisementplatform.

FIG. 3 is a diagram illustrating an example advertising ecosystem.

FIG. 4 is a block diagram of example components of a voice-controlledadvertising platform.

FIG. 5 is a block diagram illustrating example components of afoundation module.

FIG. 6 is a block diagram illustrating the example components of amanagement module.

FIG. 7 is a flowchart illustrating an example process of creating andutilizing noise signatures.

FIG. 8 is a flowchart illustrating an example process of responding to auser's voice input with targeted content.

FIG. 9 is a flowchart illustrating an example process of preloadingcommon responses.

DETAILED DESCRIPTION

A voice-controlled content delivery platform, such as an advertisementplatform, will now be described. The voice-controlled advertisementplatform, or a portion thereof, is generally deployed on auser-accessible device, such as a television, a cellular phone, avehicle-mounted device, or a kiosk, which is connected with voice inputor recording equipment and capable of outputting multimedia information.The voice-controlled advertisement platform may receive an utterance orvoice inquiry from a user regarding certain items of interest she hasobserved in content or multimedia displayed on the device or anywhereelse in the surroundings. In a first example, a user may utter “How muchdoes that cost?” when she sees a jacket on an actor in a show played ona television. In a second example, she may utter “What's on the menu?”when she is driving near a billboard right off the highway that shows animage of a restaurant. Thus, the platform allows advertisers to engagewith users in an asynchronous process that does not distract them fromthe task at hand (e.g. driving, watching a movie, reading the magazine),but ensures that users do not miss out on the opportunity to engage withthe advertising or the brand. The platform allows users to obtainadditional information when in mental states like driving where theycannot afford the additional cognitive load, or when reading or watchingtelevision when they do not want the additional cognitive load at thattime.

Advertising and other application-driven scenarios often needanticipated custom grammars, but these grammars increase the “footprint”(data storage and processing requirements), and increase the risk forpattern matching complexities. These custom grammars include customwords, custom phrases, custom sounds, etc. Prior custom grammars wererequired to be available online, or available off-line in their entiretyand fully indexed, such as for media track titles, contact names, etc.As a result, an off-line component would anticipate what applicationsmight be needed in the future and a connection component would assemblerequired grammars. These custom grammars would have an expiration timethat could expire when, for example, an ad campaign expired.

Described in detail below is an improvement whereby a platform, such asa voice-controlled advertising platform, can dynamically requestspecific speech grammar updates based on the applications it anticipateshaving to use in the near term, so that the platform allows off-linespeech recognition experiences that require no real-time network access,provide low latency recognition, and still provide custom lexiconsupport. The platform thus enables very complex custom grammars to beassembled in application/context specific chunks entirely server-side(such as for a specific ad campaign), and then pushed down to anoff-line component at the user-side—only as needed—without unnecessarilybloating the off-line component, or requiring the off-line/client-sidecomponent to do the processor-heavy work of assembling the grammars. Asa result, users have a seamless application-specific grammar, such asfor an ad campaign, without any live network connectivity, and withouthaving to load all of such grammars beforehand.

The voice-controlled advertising platform provides a server-sideinterface (website, web application, etc.) that allows an applicationdesigner to assemble custom grammars for their application or for theplatform. The platform may also assemble its own custom grammars. It maysave all user inquiries received from members of a user group andtargeted contents presented in response, where a user group can bedefined by location, by age, by profession, etc., and analyze them toidentify frequently received user inquiries and frequently presentedadvertisements. Next, the voice-controlled advertisement platform maypreload the grammars applicable to the identified user inquiries and theidentified advertisements on the device of a member of the user group,when the network load between the server and the member's device isrelatively low. When one of the identified user inquiries issubsequently received by the member's device, the inquiry may then beimmediately analyzed with one of the preloaded grammars and respondedwith one of the preloaded advertisements.

On the other hand, when a client-side application (off-line) needs tomake use of one of the custom grammars, it may ping the network whennetwork connectivity exists, and provide a set of grammar profiles thatit anticipates needing off-line. These profiles may contain links to thegrammars, which are managed server side. Each profile typically has atheme, such as “new movie releases” or “dates and times” and isassociated with a set of contextual metadata representing variousdimensions, as discussed herein. As the profile is linked to specificgrammars, the developers of the client-side application need not worryabout which words and associated grammars should be stored for specificusers or purposes at all times. Therefore, the client-side applicationneed not know anything about the grammars themselves, just the profiles.In response to a request based on a profile, the server then providesthese grammars to the client for local storage and off-line use. Throughthe use of profiles, the platform permits the developer or anapplication designer to process various contextual metadata likelocation, application usage history, value of ad campaign, etc. to becaptured in the profiles, and the corresponding grammars can bedynamically downloaded to and cached on user devices for offline usage.

For example, if a user is in Seattle, an ad is for the National FootballLeague (NFL), and the designer is targeting the campaign for a maleaudience in the 20-30 age range, then the client would request from theserver one or more profiles related to Seattle-based grammars, NFLgrammars and age/gender-based grammars. Some of the metadata that theclient sends to the server includes items or data like location, age,gender, etc. The server receives the metadata and identifies appropriategrammars from a library or database of grammars that correspond to thesemetadata—in the form of profiles—that the server then sends back to theclient. The client caches the received grammars for later offline usage.In this case, the NFL grammar may include team names, with a higherpriority for Seattle team names and Seattle player names, while the male20-30 age grammar may include a speech recognizer that handlesaudience/user responses tuned to younger male acoustical models. (Theterms “grammars” and “profiles” may at times be used interchangeablyherein.)

As discussed above, the voice-controlled advertisement platform mayreceive a spoken command from a user while she's driving, watching TV,reading this magazine, etc., such as “more info”, or any derivative ofthis command, like “more information”, “tell me more”, “how much isit?”, “Does it do X?”, etc. In the first example presented above, a usermay utter “How much does that cost?” when she sees a jacket on an actorin a show played on a television. In a second example presented above,she may utter “What's on the menu?” when she is driving near a billboardright off the highway that shows an image of a restaurant. This canhappen by pressing a steering wheel button in a vehicle, selecting abutton on a mobile device or remote control unit, using a wake-up wordsuch as “Dragon, tell me more”, or any other method to initiate theaudio capture. In response, the platform gathers contextual and otherinformation to infer the user's most likely candidate of interest. Thiscontextual/other information can include the station user is listeningto/watching, the user's GPS coordinates, a picture of advertisementthey're looking at or the audio sample of the advertisement, etc. Theplatform thus identifies the sources of the items of interest. Onesource is the multimedia displayed on or provided by the user device onwhich the voice-controlled advertisement platform is deployed, such asthe television in the first example, which might be regular programmingthat is independent of the voice-controlled advertisement platform oradvertisement presented by the platform. Another source is anymultimedia that can be observed in the user's surroundings, such as thebillboard in the second example.

Upon identifying the sources, the voice-controlled advertisementplatform may determine the items of interest, such as the jacket withpricing information in the first example and the restaurant with menuinformation in the second example, based on data indicating the user'sgeneral preferences, including basic demographic information and ahistory of interacting with the platform. Upon determining the items ofinterest, the voice-controlled advertisement platform may then presentmore information related to the items of interest in response to theuser's voice inquiry. Thus, using a series of metadata from the audiosample of the user's utterance the system infers the most likelycandidate of interest and can provide additional information to the uservia e-mail, text, physical mail, social network update, or otherasynchronous method.

As one example, the user's utterance may include certain words and/or anaudio profile such as including profanity or nasal tones. The system mayuse this data to deduce a user's emotion or sentiment, such as contemptor sarcasm, to further disambiguate the user's intent. For example, thesystem may analyze the received utterance and compare recognized wordsto a dictionary having certain words flagged with an associated emotionor sentiment, and thereby infer that the user is, e.g. angry based onuse of profanity, particularly if the system has not previouslyrecognized frequent use of profanity by this user. Further, the systemcan compare an audio profile for the user, such as a level or tempo ofthe user's utterance to a baseline profile for the user to determine afaster, higher pitched and/or louder utterance, further indicating anangry emotion for that user. By recognizing users emotion or sentiment,the system can then provide inferred data, such as delivering contentthat is “opposite” to currently perceived content if the system receivesuser's sentiment to be angry. For example, if the system detects contextinformation in the user's environment related to a political candidate,and the system detects that the user is angry, then the system deliversan ad related to the opposing political candidate.

As another example, the audio sample may include background music. Thesystem may then use this type of data to identify not only informationregarding the music (e.g. the performer or artist of the song), but alsoidentify further information on what may appeal to the user's interest.For example, by recognizing that the background music often includesOpera music, and understanding from demographic data that listeners ofOpera music are often highly educated and interested in the arts, thenthe system may provide to the user ads or other information related tolocal cultural events, lectures at nearby universities, and so forth.

As yet another example, a voice signature may be derived from the audiosample. For example, after providing information regarding a new jacketthat the user inquired about, the user can automatically purchase thejacket using the audio/voice interface, whereby that purchase isautomatically conducted by verifying the user's identity with voicebiometric authentication. Overall, such rich, targeted content tends tobe highly relevant to a user's interests without inconveniencing theuser and thus highly engaging to the user.

Unfortunately, there's often significant background noise present whenreceiving the audio sample from a user. It just requires a significantamount of processing work to filter out the background noise and applythe correct acoustical model to extract the “speech” from the “noise”.Thus, the voice-controlled advertisement platform may also establishnoise signatures to increase the accuracy of recognizing a user's voiceinquiry and the quality of a response to the voice inquiry. It maycrowd-source noise samples from different users in the same location fordifferent types of environments, such as a quiet room, a sports complex,a moving car, etc. The voice-controlled advertisement platform mayaccept voluntary submissions of noise samples, so that users would optin for this service on a voluntary basis, knowing that as they opt-in,they also benefit from improved performance from the system-.Alternatively or additionally, it may automatically and periodicallycollect background noise data by periodically sampling the backgroundnoise, extending a regular recording of a voice inquiry to also recordthe background sound alone, etc. It may then compute noise signaturesfor the various types of environments based on the noise samples andcontinue to refine the noise signatures as additional noise samplesbecome available. Next, the voice-controlled advertisement platform mayuse the noise signatures to estimate the current noise level for aparticular environment based on knowledge of the current noise level ata given location (or, if acceptable, use the noise signature when noknowledge of the current noise level is available). As a result, it mayisolate the noise from a recording of a user's voice inquiry in theparticular environment, or adjust the voice used in a response to auser's voice inquiry to account for the current noise level in theparticular environment.

Device manufacturers, such as mobile phone makers, can offer such anoise sampling feature as a background application built into thedevice, to thus increase the quality of voice input for users of thedevice. This background application would periodically or sporadicallysample background audio, and transmit that audio, along with a location(e.g. GPS coordinates) and timestamp, to network servers. The serverswould then normalize all the audio files it received for the samelocation to establish a consistent “background” noise level for thatlocation. Mobile devices could be timed such that they all report backsamples at intermittent frequencies, thereby ensuring that a relativelyrecent sample is always available to the network servers. If the mostrecent sample is too old or not current, then a later received audiofile for that location would be processed as normal, without the benefitof this sampling feature. Overall, this feature of the voice-controlledadvertising platform would allow for improved speech recognition inenvironments that traditionally perform poorly, such as stadiums,restaurants, malls, airports, trains, etc.

Various implementations of the invention will now be described. Thefollowing description provides specific details for a thoroughunderstanding and an enabling description of these implementations. Oneskilled in the art will understand, however, that the invention may bepracticed without many of these details. Additionally, some well-knownstructures or functions may not be shown or described in detail, so asto avoid unnecessarily obscuring the relevant description of the variousimplementations. The terminology used in the description presented belowis intended to be interpreted in its broadest reasonable manner, eventhough it is being used in conjunction with a detailed description ofcertain specific implementations of the invention.

Suitable Environments

FIG. 1 and the following discussion provide a brief, general descriptionof a suitable computing environment 100 in which a voice-controlledadvertisement platform can be implemented. Although not required,aspects and implementations of the invention will be described in thegeneral context of computer-executable instructions, such as routinesexecuted by a general-purpose computer or a mobile device, e.g., apersonal computer or smartphone. Those skilled in the relevant art willrecognize that portions of the invention may reside on a servercomputer, while corresponding portions reside on a client computer.Those skilled in the relevant art will appreciate that the invention canbe practiced with other computer system configurations, includingInternet appliances, set-top boxes, televisions, hand-held devices,wearable computers, kiosks, vehicle computer systems, radios, householdappliances (especially ones connected to the Internet), gaming consoles,mobile phones, regular laptops, netbooks, convertible/detachablelaptops, tablets, multiprocessor systems, microprocessor-based systems,minicomputers, mainframe computers, or the like. The invention can beembodied in a special purpose computer or data processor that isspecifically programmed, configured, or constructed to perform one ormore of the computer-executable instructions explained in detail below.Indeed, the terms “computer” and “computing device,” as used generallyherein, refer to devices that have a processor and non-transitorymemory, like any of the above devices, as well as any data processor orany device capable of communicating with a network, including consumerelectronic goods such as gaming devices, cameras, or other electronicshaving a data processor and other components, e.g., networkcommunication circuitry. Data processors include programmablegeneral-purpose or special-purpose microprocessors, programmablecontrollers, application-specific integrated circuits (ASICs),programmable logic devices (PLDs), or the like, or a combination of suchdevices.

The invention can also be practiced in distributed computingenvironments, where tasks or modules are performed by remote processingdevices, which are linked through a communications network, such as aLocal Area Network (“LAN”), Wide Area Network (“WAN”), or the Internet.In a distributed computing environment, program modules or subroutinesmay be located in both local and remote memory storage devices. Aspectsof the invention described below may be stored or distributed ontangible, non-transitory computer-readable media, including magnetic andoptically readable and removable computer discs, stored in firmware inchips (e.g., EEPROM chips). Alternatively, aspects of the invention maybe distributed electronically over the Internet or over other networks(including wireless networks). Data structures and transmission of dataparticular to aspects of the invention are also encompassed within thescope of the invention.

Related software may be stored in memory, such as random access memory(RAM), read-only memory (ROM), flash memory, or the like, or acombination of such components. Software may also be stored in one ormore storage devices, such as magnetic or optical-based disks, flashmemory devices, or any other type of non-volatile storage medium ornon-transitory medium for data. Related software may include one or moreprogram modules, which include routines, programs, objects, components,data structures, and so on that perform particular tasks or implementparticular abstract data types. As described in more detail below, theparticular tasks or abstract data types may concern internals of avoice-controlled advertisement platform or external interfaces, such assoftware development kits (SDKs), which may be used to apply thevoice-controlled platform to a specific context or extend the featuresof the voice-controlled platform.

A voice-controlled advertisement platform captures voice input from auser typically in the form of an inquiry with respect to specific visualor audio cues and presents a response to the user typically in the formof targeted content. The cues may originate anywhere, including withinthe voice-controlled advertisement platform, a device on which aspectsof the voice-controlled advertisement platform are deployed, or thegeneral environment. Referring to the example of FIG. 1, avoice-controlled advertisement platform operates in or among one or morecomputing devices, such as mobile devices 105, computer 110, set-top box112, television 113, or server 115. The computing device may be deployedin vehicles, airplanes, ships and other environments. For example, avehicle dashboard may include a computing system that listens to thedriver and delivers information in response to the driver's questionsabout anything, from vehicle radio content to information displayed on abillboard at an intersection the vehicle just passed by. The mobiledevices 105, computer 110, set-top box 112, and television 113 include anetwork card or radio or another device that enables them to communicatethrough one or more networks 140, and include audio input and outputdevices such as a microphone and speakers. The mobile devices 105,computer 110, appliance 112, and television 113 communicate via thenetwork 140 with a server 115. A data storage area 120 coupled to theserver 115 contains data pertaining to the voice-controlledadvertisement platform, and software necessary to perform functions ofthese systems. The voice-controlled advertisement platform maycommunicate with one or more third party servers 125, which are coupledto data storage areas 130. Third party servers 125 and server 115 mayshare various data, as described herein.

The mobile devices 105, computer 110, set-top box 112, and television113 communicate with each other and the server 115 and third partyserver 125 through the networks 140, including, for example, theInternet. The mobile device 105 communicates wirelessly with a basestation or access point 108 using a wireless mobile telephone standard,such as the Global System for Mobile Communications (GSM, or latervariants such as 3G or 4G), or another wireless standard, such as IEEE802.11, and the base station or access point 108 communicates with theserver 115 and third party server 125 via the networks 140. The computer110, appliance 112, and television 113 communicate through the networks140 using, for example, TCP/IP protocols. The mobile devices 105,computer 110, set-top box 112, and television 113 may also communicatewith one another over a short-range wireless standard, such asBluetooth.

FIG. 2 is a diagram illustrating an example infrastructure for utilizingan SDK for the voice-controlled advertisement platform to create a voicead. A publisher platform 202 creates a mobile application that runs on amobile phone. The publisher platform utilizes an ad network SDK 203offered by a voice-controlled advertisement platform, bundlingadvertisement functionalities offered by or provided to an advertisementnetwork 204. The ad network SDK 203 includes a voice ad SDK (e.g. NuanceVoice Ad (NVA) SDK) to create voice ads or other functionality describedherein. Speech-recognition capabilities are provided by a voice controlenvironment 206 (e.g. Nuance Voice Control (NVC)), which may be storedon local storage 207, both of which reside on mobile phones, invehicles, etc. As a result, the publisher 202 (using the SDK 203)enables a mobile phone user to use voice commands not only to accessmobile phone features but to trigger targeted content delivery.

The system of FIG. 2 and effectively bundles a voice input andassociated SDK with an ad network SDK. The voice input SDK may includeSpeechKit from Nuance Communications Inc., which initializes and managescore networking, voice recognition, and audio system components, andworks with recognition and voice output functions (e.g. Recognizer andVocalizer). Recognizer manages the voice recognition process includingaudio recording and server communication. Vocalizer defines atext-to-speech implementation, and may be initialized with a language orvoice and used for sequential speech requests. The ad SDK may beintegrated with native mobile applications and mobile operating systems(e.g., iOS, Android) and distributed via app stores to end-users. The aditself has a series of JavaScript calls that allow it to invokeSpeechKit, where the ad SDK inserts appropriate credentials required forserver conductivity and authorization. The credentials themselves may beassigned dynamically to the ad network by an authentication server.

An example of how the system in FIG. 2 operates is as follows. Based ona received trigger, the ad network 204 provides a request message to thepublisher platform 202, where the request message includes anapplication ID and an ad ID. In response, the publisher platformprovides a response message with ad content or a pointer/URL/logicaladdress to that content. A security reconciliation component 205 of thead network authenticates the ad response by communicating with anauthentication server 208. The ad network 204 may provide a request IDto the authentication server, where the request includes an ad networkID (the request ID can request a specific SpeechKit). In response, theauthentication server 208 logs the request in an event logger 210, wherethe log entry can include the ad network ID, the application ID and thead ID. The authentication server provides a response message back to thead network 204 that includes credentials, such as SpeechKit_credentials,TTL (time-to-live), and session ID data. The Speechkit credentials mayconsist of a mobile application identifier, and may be a human-readabletext string composed from an application developer's name, anapplication name, and a time stamp, which uniquely identify a particularapplication. The authentication server 208 has the ability todynamically assign these credentials at runtime, so that when fraud isdetected, the authentication server 208 can disable the fraudulentcredentials and reassign new ones, without the application developerhaving to do anything.

The publisher platform 202 sends a voice recognition request to thevoice control environment 206 on the user's device, along with log data,such as a session ID, and ad data, which the voice control environmentstores in the local storage 207. Events logged by the event logger to 10are stored in a database 212. Likewise, events, as well as localgrammars, audio files, and other data are stored in the local storage207. The data stored in both the local storage 207 and the database 212can be provided to a data warehouse to 14 for storage andreconciliation. A reporting and billing server to 16 accesses data fromthe data warehouse to provide billing to the publisher, ad network orother parties. Further, the reporting and billing server 216 cangenerate reports regarding, e.g. Ad effectiveness.

As noted herein, the system may include a scripting tool that has theability to process metadata such as gender, location, or any other datasent back to the system from the client along with the utterance, ordetermined from the utterance by the system, to help determine whichbranch of a previously scripted dialogue to move to. For example, if theuser is asked in the system what her favorite food is, in addition tothe received response, the system knows that the user is female based onthe utterance, and the script may respond with a different response toher than if the user was male. This scripting tool provides addedintelligence in the execution of a script, thereby leading to dialoguesthat can be highly responsive to user preferences. It also provides aneasy means by which a lay-developer, rather than a sophisticateddeveloper or a computer programmer, can create a preprogrammed scriptfor providing targeted content to a user based on the various inputdescribed herein.

The voice-controlled advertisement platform, such as that shown in FIG.2, would support an advertisement ecosystem, as illustrated in FIG. 3.In today's digital world, advertisers and advertising agencies on thedemand side 305, such as Toyota and Digitas, would like to advertisevarious products and services offered by the advertisers through onlinesocial media and other digital channels. On the other hand, hosts ofthese digital channels, and application developers for these digitalchannels, on the supply side 315, such as Facebook and Angry Birds,would like to utilize advertisements of various products and services assources of revenue. The role of advertisement technology partners 310,which produce advertisement tools and manage advertisement networks, isto work with the demand side to implement the desired advertisements anddeliver the results to the supply side. Examples of such advertisementtechnology partners 310 include Celtra and Millennial Media. Thevoice-controlled advertisement platform described herein may be adoptedby advertisement technology partners 310 to bring intelligentadvertising to targeted consumers.

Voice-Controlled Advertising Platform

FIG. 4 is a block diagram illustrating example components of avoice-controlled content delivery or advertising platform 400. Thevoice-controlled advertising platform 400 accepts different types ofinput to help determine what a user is communicating through the user'svoice and respond to the determined user communication with appropriatecontent or advertisement. The terms “content” and “advertising” aregenerally used interchangeably herein, and many examples described thedelivery of advertising, but those skilled in the relevant art willreadily recognize that any form of content may be used, not justadvertising. The input to the voice-controlled advertisement platform400 includes audio input, including the foreground user input andbackground noise data, non-audio input, including user data, media dataand environment data, and raw advertisement data, includingadvertisement material and conversation data. The output from thevoice-controlled advertisement platform 400 comprises targetedadvertising or content.

The voice-controlled advertising platform 400 is configurable for manyenvironments. It can be installed entirely on a server computer, likethe server 115, it could be deployed to another device, such as themobile devices 105, computer 110, set-top box 112, television 113, or itmay be distributed among one or more devices and a server or amongmultiple devices in a peer-to-peer arrangement. The voice-controlledadvertising platform 400 includes a foundation module 410 and amanagement module 430, which while shown as two separate modules, couldbe combined as a single module on a single device, or divided among twoor more modules on two or more separate devices. The advertisingplatform 400 accesses and/or stores data in a text-to-speech database440, a speech recognition database 445, a user database 450, an rawadvertisement database 455, a conversation database 460, a mediadatabase 465, a noise signature database 470, an environment database475 and a targeted content database 480.

The user input is input data from a user typically in the form of aninquiry with respect to specific media data. In some implementations,user input includes audio data representing words spoken by a user andpossibly a corresponding transcript of the spoken words. Thevoice-controlled advertisement platform 400 may designate specialphrases, such as “more information”, to represent an explicit userinquiry. However, it may accept any other user utterance and extract auser inquiry with further analysis.

In some implementations, user input is submitted by a user into thedevice on which the voice-controlled advertisement platform 400 isdeployed. In some implementations, a microphone of a device, such as amobile device or a video game console, can record and transmit to thevoice-controlled advertisement platform a recording of words spoken bythe user. The voice-controlled advertisement platform 400 can alsoreceive raw or processed user input from another device that capturesthe user input. For example, deployed in a television, thevoice-controlled advertisement platform 400 may receive audio data froma mobile device associated with a user or from a set-top box. It mayalso process original user input to derive further information.

The noise data includes general ambiance noise. It could correspond todifferent types of environments, such as a quiet room, a supermarket,and a sports stadium, and different types of noise may be collected andanalyzed in different ways. However, the noise data represents the soundin the background that is to be isolated from the sound in theforeground, which is typically the user input. Like the user input, thenoise data may be collected in similar ways. It can be transmitted orsubmitted as standalone data or as an extension of a normal recording ofthe user input.

The user data represents or describes the user who is interacting withthe voice-controlled advertisement platform 400. It is useful fordetermining what type of advertisement to present to the user and how.The user data can include biometric information, which can be captured,for example, using facial imaging technology, infrared sensors, and thelike. It may also include public records, such as marriage licenses,property documents, and law court proceedings. It can further includevarious types of information characterizing a user's interests. As oneexample, a user may provide the user's demographic data, such as gender,age, birthday, and birth place. As another example, the user may alsoindicate the user's profession and hobbies as well as generalpreferences and particular items of interest in different categories. Asadditional user data, a user may provide the user's online or otheridentifies, such as an email ID, a penname, a stage name, and otherpseudonyms, as well as the user's speaking and writing samples, whichcan be further analyzed.

Furthermore, the user data can include information related to a mobiledevice operated by the user, such as the phone number of a mobile deviceor an IP address of a desktop computer, or information describing adevice used to present media content to the user, such as informationdescribing an operating system of a laptop computer. It may include datatransmitted as part of a user's online activities, such as the user'stweets and Facebook posts, and data tracking a user's online activities,such as Internet cookies and logs. Moreover, the user data may includeadvertisements that a user previously viewed and the user's response topreviously-presented advertisements during the user's interaction withthe voice-controlled advertisement platform 400.

In some implementations, the voice-controlled advertisement platform 400collects user data directly from users through surveys orquestionnaires. In some implementations, a device in which thevoice-controlled advertisement platform 400 is deployed gathers userdata using a radio antenna, microphone, camera, and so forth andtransmits it to the voice-controlled advertisement platform. In someimplementations, the voice-controlled advertisement platform 400receives the user data from remote sources, such as a server computer.In some implementations, the user data is received from a deviceassociated with a user, communicated via a wireless standard, such asBluetooth, WiFi, or Near Field Communication (NFC). The voice-controlledadvertisement platform 400 may also gather user data during itsinteraction with the user.

The media data represents or describes the media content presented to auser by a device on which the voice-controlled advertisement platform400 is deployed. The voice-controlled advertisement platform 400 usesmedia data for a number of reasons, including for interpreting the userinput and for determining how to deliver targeted content oradvertisement to the user. Depending on how the voice-controlledadvertisement platform 400 is deployed, the media content can be one ormore ads, music, radio, a movie, a television show, a televisioncommercial, a radio commercial, a video game (on a console, computer,mobile device, etc.), online video, streaming audio, a banneradvertisement, kiosk images, and so on. The media data may also includeplayback information related to the media, such timing data related tothe playback of a movie and other metadata associated with the media. Insome implementations, the media data includes a log that describes whereor when content elements appear in a media item. In general, the devicewould transmit the media data to the voice-controlled advertisementplatform 400 for analysis automatically or in response to a request bythe voice-controlled advertisement platform 400.

The environment data represents or describes items related to a user'slocation and/or in a user's surrounding that are likely to be of theuser's interest. For example, the environment data can include the GPScoordinates of the user, possibly with additional data such as velocity,altitude, etc. Additionally, when the user is at home, an item could bea dog that sits by the user or a song the user was listening to; whenthe user is in the office, an item could be a book on the user's deskthat the user was reading; when the user is on the road, an item couldbe a store sign or a billboard the user just passed by. As the itemdepicted by the environment data could be the source or an inspirationof a user's inquiry, the environment data enables the voice-controlledadvertisement system 200 to better target advertisement to the user'sinterest in response to the user input.

The environment data can be captured in various ways. In someembodiments, a user can supply a video steam of the user's surroundingswhich include images as well as sounds. In some embodiments, a user cansupply GPS coordinates of the user's current location via the user'smobile device, and the voice-controlled advertisement system 200 canwork with external systems to determine the items that are present atthe coordinates (e.g. restaurants near to a sign/billboard).

The advertisement material describes the content in the targeted contentpresented to the user, either from an external ad source or by thevoice-controlled advertisement platform 400 in response to the user'sinput. In some implementations, it includes a group of voices that is tobe used to synthesize speech for conversational advertisements. Ingeneral, the advertisement material includes text, images, audio, video,and a combination of those, and so on regarding various items that couldbe interest to a user. It may include a static presentation or it maycontain information that prompts a user to take further actions. Theadvertisement material is normally supplied by advertisers.

The targeted content is responsive to the user input, such as questionor a request. The targeted content may include text, audio, video,pictures, statistics, a link to a file stored in a server computer, andthe like. In some implementations, the targeted content includes basicuser prompts and responses. In addition to information that specificallyaddresses the user's inquiry, the targeted content may containadditional information about the subject matter of the user's inquiry orinformation about related subject matter, which may trigger furtherinquiries from the user.

The targeted content may be delivered to the user through a device onwhich the voice-controlled advertisement platform 400 is deployed,through another device being used to consume media content, or via othermeans. As one example, when it is deployed on a television, thevoice-controlled advertisement platform 400 can send the targetedcontent to the user via email or text, or to the television in adisplayed informational window. As one example, even when it is notdeployed on a user's desktop computer, the voice-controlledadvertisement platform 400 may deliver the targeted content to a userthrough a web browser installed on the user's desktop computer, whichthe user can access at the user's convenience. FIG. 5 is a block diagramillustrating example components of the foundation module 410. Thefoundation module 410 comprises a data transfer module 510, a speechrecognition module 520 and a text-to-speech recognition module 530.

The data transfer module 510 receives all types of input data. It sendsthe input data to appropriate modules for immediate processing, theresults of which would be saved into appropriate data storage ordatabases, or stores the input data into appropriate databases forfuture processing. For example, the data transfer module 510 would sendthe noise data and the user input to appropriate modules for immediateprocessing and saves the results into the noise signatures database 475and the speech recognition database 445, respectively. As analternative, it may store these data into appropriate databases first.It would also save the user data, the media data, the environment data,the conversation data, and the advertisement data respectively into theuser database 450, the media database 465, the environment database 485,the conversation database 460, and the advertisement database 455.

The speech recognition module 520 receives audio data and generates atranscription of spoken words represented by the audio data. The audiodata includes the user input that is received by the voice-controlledadvertisement platform 400 which represents words spoken by the user.The speech recognition module 520 utilizes a speech recognitionapplication, such as Dragon, which is also sold by NuanceCommunications, Inc. The speech recognition module 520 stores thetranscription data in the speech recognition database 445.

In addition, the speech recognition module 520 may build or receivespecialized grammars. It manages a lexicon of textual objects and buildsor accesses language models to map speech data to the textual objects.The lexicon could be for a specific event or scenario. For example, fora music concert by a pop artist, the lexicon may include the name of theartist, the location and play list of the concert, and some otherjargons used by a young population. The focus of the lexicon normallyreduces the size of the specialized grammars, making them especiallysuitable for offline use in the corresponding event or scenario. Thesystem may dynamically grow the language models as it processes morespeech data. For example, as the number of user inquiries containing aspecific name increases, it may build a language model that includes acomprehensive and accurate sound characterization of the name and ofcertain attributes usually associated with the name. More generally, theuser inquiries can be compiled in a crowdsourcing context. For example,when an ad campaign is running, a large number of people could engage inthe one or more ads included in the campaign. The results are userinteractions associated with various ad categories and user groups,which enable the system to identify the most relevant dialogues for eachof the ad categories and user groups and build specialized grammarsaccordingly. In addition, the speech recognition module 520 may maintaina master set of language models based on speech data received acrossmultiple client devices on a server computer or in a cloud environment,and synchronize the language models used on individual devices with themaster set. Finally, even being built from crowd sourcing for a specificevent or scenario, a language model may not account for all possibleuser interactions. In the actual deployment, a user may still deviatefrom an anticipated set of inputs. Therefore, the system may alsoinclude a default set of responses in the language model to conclude adialogue in an acceptable manner. For example, the system may include adefault ad to provide if the system is unavailable or otherwise cannotaccurately serve up a targeted ad to the user.

Furthermore, when the voice-controlled advertisement platform isdeployed on both a user device and a server computer, the speechrecognition module 520 may preload or receive via the network some ofthe specialized grammars or language models, which include the defaultset of responses, to client devices to reduce response time to a user.In some implementations, the speech recognition module 520 may assign aweight to each language model depending on how often it has been used,how much it has expanded, how complex it is, and other factors. It mayassign a larger weight to a language model that is more useful, morerefined, or more time-consuming, for example, and it may assign a singleweight to each language model or multiple weights respectivelycorresponding to predefined user groups.

Next, the speech recognition module 520 may preload those languagemodels having larger weights to all the client devices or to the clientdevices of members of individual user groups. The preloading may beperformed periodically or at specific times. It may be performed whenthe network load between the server computer and the client device isrelatively low, when the weights of the language models have changedsufficiently, when sufficiently many new language models have beenbuilt, and so on. The preloaded language models may then be stored in alocal storage or in a cache on the client device to further expedite theutilization of these language models. As a result, a user inquiry may beanalyzed immediately if one of the preloaded language models isapplicable.

The text-to-speech module 530 receives text and converts it into speech.The text-to-speech module 530 may receive predetermined conversationdata to convert to speech. For example, the text may be a response ofthe voice-controlled advertisement platform to a user. Thetext-to-speech module may also receive input pertaining to speechpreferences. The text-to-speech module 530 utilizes a speechsynthesizer. One suitable speech synthesizer is sold by NuanceCommunications, Inc. and called Vocalizer. The text-to-speech module 530stores the speech data in the text-to-speech data database 540. Ingeneral, no new components would be required on the client device, sincemuch of the technology resides on the server, and is accessible to theclient via the network, although as noted herein, the client may includesome locally stored data or components, such as specific grammars foruse in an off-line context.

FIG. 6 is a block diagram illustrating the example components of themanagement module 430. The management module 430 comprises a noisesourcing module 620, a user input analysis module 610, an interestelement identification module 630, and a targeted content determinationmodule 640. These components can be used to improve audio recognition,and improve user advertisement interaction experiences.

As discussed above, the voice-controlled advertising platform 400 isconfigurable for many environments, and thus the components discussed sofar may be deployed on a server computer or one or more devices. Oneapplication typically deployed on the client side may account forinterruptions as speech or advertisements are being played back to theuser. Specifically, the voice-controlled advertising platform 400 mayemploy existing technology (e.g. Nuance's “Barge-In” technology) tocontrol the playback—pause, stop, fast forward, fast backward, changevolume, etc.—in response to the sound levels in the playbackenvironment. It may further utilize voice signatures or other speechcharacteristics to distinguish the foreground sound from the backgroundnoise. As a result, when a user is in a crowded environment, when agroup of people start talking in the background, the voice-controlledadvertising platform 400 may amplify the playing of an advertisement,while when the user starts speaking, the voice-controlled advertisingplatform 400 may interrupt the playing of the advertisement, to ensurethe best user experience of enjoying the advertisement.

Noise Characterization and Improving Audio Recognition/Delivery

Considering first how the management module 430 improves audiorecognition/delivery and noise characterization, the noise sourcingmodule 620 receives noise data from or corresponding to different typesof environments. The noise sourcing module then generates a noisesignature for each environment or type of environment, and stores thenoise signatures in the noise signature database 475 for subsequent use,including sharpening foreground audio (e.g. “voice”) corresponding tothe same environment. The noise sourcing module may receive multiplenoise samples from multiple users in various environments, normalize oraverage the samples, and generate a noise signature with respect tovarious sound attributes, such as loudness, intensity, pitch andfrequency. It typically accepts noise data received from a large numberof sources at a particular location and time, and iteratively refinesthe noise signatures based on the enriched noise data. In someimplementations, the noise data is obtained from leaving a microphoneexposed after recording an input from a user or before playing aresponse to the input to the user. Alternatively or additionally, thenoise data comes from voluntary submissions from users. For example,functionality stored on mobile phones may allow users to opt-in toperiodically or sporadically provide to the system background noisereceived from one or more microphones on the mobile device, which thesystem uses to improve, for example, voice recognition, as describedbelow.

Additional features of a mobile device and of existing APIs may berelied on to automatically gather data. For example, granular locationdata obtained through location detection features, such as Qualcomm'sGimbal platform or Apple's iBeacon solution, may be leveraged totriangulate background noise on one side of a building as opposed toanother. For example, multiple mobile devices in a location may eachgather background noise, and through the use of iBeacon (using Bluetoothsignaling), identify the location of a noise and locations of userswithin that location.

Further, using a combination of sensors on a phone, such as a lightmeter, a gyroscope, and an accelerometer, the system can make aninference as to whether the phone is “out” (on the table, in the hand)or “in” (in a pocket or inside of a purse). For example, if the lightsensor does not detect light, the accelerometer detects continuousmovement, the GPS sensor does not detect geographic movement, and themicrophone senses loud background music, the system may determine thatthe user could be dancing. Knowing this level of phone positioning orcontext and further correlating the background noise among all thephones in a given area that are in the same position (in pocket, inpurse, outside) will help strongly identify the specific signature ofthe background noise at that moment in time to help remove it entirelyfrom the signature of the user who is speaking into the system.Typically, the noise data gathered as described above can easily beretrieved from the phone, and then transmitted to the server.

From this data, the noise sourcing module 620 can predetermine or definedifferent types of environments, such as a quiet room, a supermarket, anairport, restaurant, and a sports stadium, and it may allow users todefine additional environments. It may determine the environmentassociated with each piece of noise data using information explicitlyprovided by the user or work with external systems to determine thelocation of recording. For example, each received noise signal caninclude a timestamp and location coordinates (e.g. GPS latitude andlongitude), which the system can then use to identify a business locatedat those coordinates (e.g. an airport, a stadium, a restaurant, etc.).

In general, a noise signature for an environment, which represents theaverage noise level, may be used to estimate the current noise levelwhen the current noise level for the environment is not available. Insome implementations, when the voice-controlled advertisement platform400 is to extract a user's voice from a recording performed in anenvironment, the noise sourcing module 620 may use the noise signaturefor the environment to isolate noise from the recording for the speechrecognition module 320 to properly recover the user input usingavailable noise subtraction techniques. Alternatively or additionally,before the voice-controlled advertisement platform 400 sends a responseto a user in an environment, the noise sourcing module 620 may use thenoise signature for the environment to estimate the current sound levelfor the environment or to gauge how the current sound level that isavailable compares with the average noise level for the environment. Thenoise sourcing module 620 may then communicate the result to thevoice-controlled advertisement platform 400 to select an appropriatevoice for delivering the content or response, or even whether to sendthe response or send it in a different fashion (e.g. via e-mail). It mayalso halt further action until the current sound level is significantlylower than a predetermined noise threshold or noise level indicated bythe noise signature.

FIG. 7 is a flowchart illustrating an example process performed by thenoise sourcing module 620 for creating and utilizing noise signatures.At block 705, the noise sourcing module 620 receives and stores audiodata with background sounds as noise data for various environments. Forexample, it may capture, from each of multiple users, the backgroundsound after a user provides a voice input or before providing a responseto a user's voice input. The received audio data may include not only atimestamp indicating when the audio data was obtained at each user'senvironment, but also location coordinates associated with thatenvironment.

At block 710, the noise sourcing module builds a noise signature foreach location or type of environment, such as a quiet room, asupermarket, and a sports stadium. For example, the system gathersmultiple noise data samples and combines or normalizes them when certainthresholds are met to create a noise signature file. For example, thesystem may employ a time threshold and a location threshold and onlycombine data samples within the time and location thresholds (e.g.within the last hour, and within a predetermined radius distance ornumber of decimal degrees/latitude-longitude seconds). Once processed,the system can store the processed audio as a noise signature file forthat location or for similar locations that correspond to one type ofenvironment having a substantially similar noise signature, and providean environmental tag associated with that location. For example, thesystem can perform a query of a location index using the locationcoordinates to determine the type of business existing at that location,and then tag that noise signature file with metadata indicative of thattype of business.

In processing a newly received audio portion or recording containinguser input in an environment of a specific type, at block 715, the noisesourcing module 620 determines whether any noise signature file for thespecific environment exists. If the noise signature file exists, atblock 720, it subtracts noise data from the recording to recover userinput. In other words, if the newly received audio is from or associatedwith a location for which the system already has stored a noisesignature file, then the system assumes that the noise at that locationis substantially similar to that associated with a noise signature file,and thus the system can subtracts that noise data from the receivedaudio to obtain the user input. The system may also check in block 715whether the noise signature file has a timestamp that is sufficientlyrecent. For example, the system may only perform the functions underblock 720 if the noise signature file for that location is less thaneight hours old. (If the noise signature file is beyond the timethreshold, then the system will again attempt to create a new noisesignature file as noted above.)

Otherwise, at block 725, the system uses a noise signature previouslybuilt for the specific type of environments to estimate the noise datafor the specific environment. For example, the system determines thelocation associated with the newly received audio portion and determinesa type of business or other environment associated with that location.The system then retrieves a noise signature tagged with that type oflocation, and then processes the received audio based on that type ofenvironment.

The noise sourcing module 620 also determines a type of ad or responseto provide to a user. In producing a response to a user in a specificenvironment of a specific type, at block 730, the noise sourcing module620 similarly determines whether any noise signature file for thespecific environment is available. If the noise signature file isavailable, at block 735, the system uses the noise data from that fileto determine whether to include a voice component in the response andthe attributes of the voice component. For example, when the noise levelis expected to be high in that environment, then the response voiceprovided is louder. Alternatively, if the noise level is expected to below, the response provided is likewise lower. If the noise signaturefile is not available, at block 740, it uses a noise signaturepreviously built for the specific type of environments to estimate thenoise data for the specific environment, Further, if the noise is simplytoo high, then the system may employ alternate channels for deliveringthe content, such as by e-mail, text, etc.

An example of a generic data structure that can be used for the noisesignature file can be as follows:

Environment File ID Time Location Type Noise Signature 12345678 2:23:1347.606201; Shopping Noise_File_A −122.3321 mall 23456789 5:24:5647°12′31.885″ Café Noise_File_B 122°50′28.798″

As noted, the noise sourcing module 620 receives noise datacorresponding to different types of environments, generates a noisesignature for each location or each type of environments, and stores thenoise signatures in the noise signature database 475 for subsequent use.The system can perform some post-processing to noise signature files tohelp tune or optimize the system. For example, the system can comparenoise signature files from the same location over a time interval (days,weeks, etc.). If the system determines that the noise signature file issubstantially equivalent within that interval, then the system canautomatically apply the stored noise signature file for that location orthat environment, and no longer need to request noise data to beperiodically provided by user devices at that location.

In gathering background noise, the system can also employ techniques toimprove its performance, such as providing prompts for the userregarding the environment. For example the system can compare a currentnoise signature to stored noise signatures, and if the current noisesignature is substantially different from all storage signatures, thenthe system can prompt the user with one or more queries to inquire aboutthe nature of the noise at the user's location. In one embodiment, thesystem initiates an instant messaging or SMS/IMS session with the user'smobile device to obtain further information regarding the user'slocation and noise there.

Characterizing noise may be helpful for recognizing or eliminating noisein other, non-advertisement-related contexts. In general, noisesignatures could be useful whenever input audio data is not created inan ideal recording environment. Thus, the system may be integrated withmany different environments, and preferably installed below anapplication layer so as to intercept APIs for certain audio functionsand provide improved performance. Thus, the system may be logic insertedright after audio data is received from a microphone.

As one example, any automated telephone customer service (interactivevoice response (IVR) system), which expects user speech made in anylocation, may benefit from the use of noise signatures. Thus, the systemcan work with or intercept APIs from the IVR to process audio receivefrom the user before that audio is provided to the IVR system. Asanother example, an application like Shazam, which expects music orother audio entertainment playing in any location, may also benefit fromthe use of noise signatures. Specifically, Shazam may receive audioinput that contains a replay of a recorded song as well as backgroundnoise. By allowing the system to intercept the audio data and apply anappropriate noise signature to eliminate or attenuate the backgroundnoise, the Shazam application can receive the song audio with minimaladded noise, and increase the accuracy of identifying the recorded song.

The system can also be used to perform automated content recognitionthat extends beyond simply recognizing a user's utterance. As oneexample, the system may be used to monitor background audio to detect,e.g., an advertisement and perform an analysis of that advertisement,such as comparing it against a database to identify the particularadvertisement that a user is currently listening to, in a manner similarto how Shazam analyzes audio, compares it to a database and determineswhich song is being played. As another example, the system may becombined with existing technology, such as those which manage televisionprogramming information and those which perform content discovery ontelevision shows (e.g., processing words spoken in television shows orcorresponding closed-captioning text), to periodically determine thecategory, genera, or even plot of a television show a user is watching.Based on information from various sources indicating the user'spreferences, such as being combined with the user's internet searchhistory data, the system may be able to notify the user when certaintopics of the user's interest are being discussed on one of thetelevision shows often viewed by the user, so that the user can watch itor record it. As discussed above, such automated content recognition canhelp further distinguish between the background sound and the foregroundsound on the one hand and narrow down the user's areas of interest onthe other hand, thus improving targeted content delivery.

For example the system may use Nuance's Dragon Mobile Assistant toidentify, via automatic content recognition, that the user is watchingthe television station CNN. It may further correlate from his searchhistory on the mobile assistant that he is interested in technologystocks. Combining this information with personal profile data, apersonal assistant on the user's mobile device might notify the userthat there is a breaking-news story on technology stocks on CNN, even ifthe user is not watching CNN at that time. Under this example, thesystem may integrate closed captioning data for the CNN channel orintegrate with a third party vendor such as boxfish.com.

In addition, the system may compare and contrast the noise signaturesfor different environments and determine appropriate prompts to the useraccordingly. For example, the system could listen for noise signaturesfor freeway driving vs. city driving, and make assumptions about thedriver's ability to focus/participate (more distracted in city driving,more relaxed in freeway driving, etc.). As a result, the system couldengage a more soothing dialogue when it recognizes that the driver is instop-and-go type driving (city or in traffic on a freeway), be moreverbose or entertaining when it recognizes that the car is on thefreeway, and so on.

The system may also develop noise signatures not only for certainenvironments, but also for certain situations, so as to provideadditional feedback to users or to implement other functionality. Noisesignatures can be useful in recognizing certain noises themselves,especially when the noise level is higher than normal, which couldsignify an issue or abnormality in the environment. For example, it maybe especially loud when one or more windows of a car, including asunroof, are not fully closed. In this case, the noise level in the carmay be significantly greater than that represented by the noisesignature for a sealed car and more comparable to that represented bythe noise signature for a car with open windows or a moving motorcycle,for example. As a result, the system may recognize the noise ascorresponding to a noise signature file for a moving car with one ormore open windows, and alert a user in the car of the issue and prompthim to take remedial actions to resolve the issue. For example, thesystem may provide a speech output to the vehicle's sound system, orsend a text to the user's mobile device, asking the user to closewindows in the car, particularly to improve voice recognition if thesystem has just received an utterance from the user for the system toanalyze. Thus, the system compares received audio data to noisesignature data to understand certain environments and associate withthose environments user prompts or queries that the system canautomatically provide to the user when those particular noise signaturesare detected. Furthermore, the noise signatures may be useful inprocessing received audio to subtract out noise and determine context orenvironmental data associated with the user. For example once noise issubtracted from received audio, the system can determine that twodistinct types of audio data remain, namely an utterance from the user,and a background song. Based on these two types of remaining audio data,the system may be able to determine with more certainty whether theutterance relates to the song. Thus, the system can more accuratelydetermine the user's inquiry and intent or desired result.

Thus, the voice-controlled advertising platform can identify uniquenoise signatures or audio footprints for received audio to identify notonly certain environments to be identified, but also additional content.That additional content can be the spoken input from a user, where thatspoken input can be used for user authentication. Thus, the system canreceive user input, determine a voice signature from that user input,and compare it to a database of other user signatures to automaticallyauthenticate a user. Thus, the system can use a voice biometric toautomatically identify a user and to authorize or authenticate anoperation. For example, by using such a voice biometric, the user canmore easily, yet securely, authenticate a credit card purchase or othermerchant or financial transaction.

Another application in noise recognition is advertisement frauddetection. Fraud associated with ad serving may be caused by botalgorithms which generate fake clicks or devious publishers who “stackads” so that only one ad is visible to a user and yet one click countsfor multiple ads. Especially in the case of bot algorithms, the fakeclicks are performed through software manipulation, thus resulting in noclicking sound at all, often in the absence of humans. As a result, thefake clicks can be associated with ambient noise instead of acombination of mouse clicking and other human activity. Accordingly, byusing ambient noise detection combined with background noise signatures,the system can validate genuine engagement with the user in ad serving.

Moreover, by delivering voice ads to a user's device as describedherein, and then monitoring for delivery of those ads via that device,or other nearby devices, a “feedback loop” is provided so that thesystem can listen for the delivered ad and confirm that it has beendelivered. For example, the system may determine that a beer ad shouldbe delivered to a particular device or user. The system can then monitoror “listen for” delivery of that ad within that geographic region, suchas monitoring background audio from nearby devices to confirm that theintended beer ad has been played within that geographic region (even ifit is uncertain of the specific device that played the ad).

Improving User Advertisement Interaction Experience and Content Delivery

In addition to improving audio recognition and delivery, the managementmodule 430 can also improve user advertisement interaction experiencesand content delivery, which includes selecting particular advertisementsbased on the user and information regarding her surroundings or othercontext. Further, the management module can implement advertisingscripting tools, and cash speech recognition grammars and commonresponses, as described below.

The user input analysis module 610 examines the input from a user tohelp determine which targeted content to present to the user. In someimplementations, the user input analysis modules predefines a set ofphrases, such as “More information” and “Tell me more about [to befilled in by a user]” and processes only those phrases as the userinput. For that purpose, it may build custom grammars and parse the userinput accordingly. In some implementations, the user input analysismodule 610 receives the user input from the speech recognition module420 and attempts to identify an inquiry from the user input. The userinput may or may not be specific enough to identify the subject matterof the user's inquiry. For example, when watching a comic show featuringa standup routine, a user may be wondering who the comedian is. As aresult, the user may utter “Who is that?”, which is very generic, or“Who is that comedian?”, which specifically refers to the subjectmatter. In this case, developers employing the system can dynamicallycreate or access custom grammars to support a range of comedians,actors, or other entities that may be on screen, based on a live feed ofprogramming information from the TV network, (or closed captioning forthat live feed) for example. The user input analysis module 610therefore analyzes the speech recognized by the speech recognitionmodule 420 according to the customer grammars to extract any usefulinformation, using various natural language processing techniques. Whenit deduces the subject matter of the user inquiry, the user inputanalysis module 610 would pass it on to the interest elementidentification module 630.

The user input often would not reveal enough information to preciselydetermine the subject matter of a user's inquiry, or the interestelement. When that is the case, the interest element identificationmodule 630 analyzes different types of data in an attempt to determinewhat the interest element is. One source of interest elements is themedia data transmitted through the device on which the voice-controlledadvertisement platform 400 is deployed. Another source is theenvironment data characterizing a user's surroundings, which oftencomprises media contents, such as streamed videos of a user'senvironment. The initial environment data may also lead to furtherenvironment data which comprises media contents. As one example,depending on approximately when and where the user input is received oraccepted, the interest element identification module 630 may accessinformation on what was playing on a user's radio or television at thattime. It may recognize some audios or videos being displayed on theuser's radio or television using voice biometric or image matchingtechnologies and compare the recognized audios or videos with knownprogramming lineups. As another example, the interest elementidentification module 630 may use GPS coordinates received from the userdevice to retrieve maps and images of the corresponding location with alist of points of interest. Yet another source of interest elements isthe advertisements which were previously presented to a user and towhich the user then responded. These advertisements would also containmedia contents. Any source data may be limited to those to which a useris exposed during the last thirty seconds or a specific timeframe thatis commensurate with an average person's response time.

Given potential sources of interest elements, the interest elementidentification module 630 needs to narrow down potential interestelements. There may be various items in specific media contents that canbe candidate interest elements. Such items include a visual object, asound, a spoken word, a graphic, or the like, that is displayed, played,referenced, or otherwise presented to a user as part of media content.For example, it could be the image of a car or the utterance of “thecar”. In general, the interest element identification module 630analyzes the specific media contents to recognize items that are likelyto be interest elements using various techniques, such as object-basedimage analysis (OBIA) and music structure analysis. In someimplementations, media metadata, including a list of potential interestelements, is predefined and received by or delivered to thevoice-controlled advertisement platform 400 from external sources. Amedia provider may supply certain media contents together with metadata,which constitutes annotations or translations of the specific mediacontents for further analysis, or explicitly enumerates potentialinterest elements and their times of appearance in the specific mediacontents. For example, the platform 400 may access a database ofbillboard locations and content displayed on those billboards from abillboard company like OutdoorBillboard.com, and analyze a user's querybased on content on a billboard at a location near to where the usermade the query. As another example, the platform 400 may access adatabase of radio content broadcast by one or more stations at aparticular time and location, and compare that to audio content receivedfrom the user (e.g. accessing a radio content database from ClearChannel). Thus, users may have a conversational experience with an adthey hear on the radio. In some implementations, the interest elementidentification module 630 analyzes the specific media contents to deduceinformation possibly related to interest elements, using naturallanguage processing, image matching, text analysis or other techniques.

When the specific media contents are complex with many recognizableitems, however, the interest element identification module 630 mayanalyze the user data, which often reveals the identity and interests ofa user, to help narrow down the list of candidates, using various datamining and correlation techniques. As one example, the user data mayindicate that a user listens to a lot of rock music, does not read manybooks, has no interest in cooking, and has an extensive collection ofwatches. The interest element identification module 630 may then use theuser data to favor items related to rock music and watches and disfavorthose related to books and cooking. As another example, the user datamay indicate that a user has repeatedly inquired about arts and craftsduring the user's online activities or the user's interactions with thevoice-controlled advertisement platform. Therefore, the interest elementidentification module 630 may give more weight to a recognizable itemthat is related to arts and crafts.

Even after recognizing a potential interest element, the interestelement identification module 630 may need to conduct further analysisto identify the interest element with greater probability. For example,given the image of a car, the interest element identification module 630would look it up in a car database to get the make and model of the car.Such study can be conducted through the internet, external knowledgebases, external knowledge-discovery specialists, and so on.

Once the interest element identification module 630 determines an itemof interest, the targeted content determination module 640 determineswhich targeted content (including targeted ads) to provide to a user inresponse to the user's input and how to deliver the targeted content tothe user. When the user requests additional data, e.g. responds to anad, the user's response may become additional user input, and thetargeted content determination module 640 responds to the additionaluser input.

The targeted content determination module 640 chooses the targetedcontent from the advertisement database 455. In general, it identifiesan advertisement or additional content that responds to a user's inquiryand thus is related to the interest element. For example, if the user'sinquiry is “How much is that car?” with respect to a car of a specificmake and model featured in a scene of a movie the user is watching, thetargeted content would include pricing and related information of thatspecific car. In some implementations, the targeted contentcorresponding to an item in specific media contents may bepredetermined. For example, a media provider may supply certain mediacontents with related advertisements and metadata specifying potentialinterest elements in the media contents and the correspondingadvertisements.

FIG. 8 is a flowchart illustrating an example process of responding to auser's voice input with targeted content. At block 805, the user inputanalysis module 410 receives audio input from a user and analyzes userspeech in the audio input to recognize an inquiry. In one example, theuser may be driving in a car, passing a billboard, and requests moreinformation regarding a product displayed on the billboard by saying“tell me more about that billboard”. The user's smart phone, a dashboardcomputer in the car, or a combination of the two, receive the audioinput from the user requesting more information. The phone/car addsmetadata to the audio input and sends both the audio input and themetadata to the server wirelessly, where the metadata includes useridentification information (e.g. mobile/car ID number), time of day,location of the car/phone, etc. Upon receiving this, the systemrecognizes the inquiry from the user based on the utterance and invokesthe modules under the management module 430.

At block 810, the interest element identification module 630 identifiessources of the interest elements, namely the subject matter of theuser's inquiry, from media data provided by a device on which thevoice-controlled advertisement platform 400 is deployed, environmentdata regarding the user's environment, and/or the targeted contentpreviously provided or displayed by the voice-controlled advertisementplatform 400 as the response to the user's last inquiry. In this manner,the system is being receptive to a user's inquiring about a wide rangeof items to which the user is exposed. Continuing with the aboveexample, the system accesses a third-party database to determine thecontent of the ad displayed on the billboard based on the user'slocation and time.

At block 815, the interest element identification module 630 determinesone or more interest elements from the one or more sources identified inblock 810. Under block 815, the system may also use user data showingthe identity and interests of the user and the user's previous inquiriesunrelated to recent, previous inquiries, such as for deriving moredetail regarding a user's interests. Again, continuing with aboveexample, the system analyzes the users inquiry, which specificallymentions a billboard, and thus the interest element identificationmodule 630 determines with high probability that the user is interestedin additional content regarding a product displayed on the billboardthat the user recently passed in his car. An identity of that particularproduct can be obtained from the third party billboard site.

At step 820, the targeted content determination module 640 presents orprovides targeted content that is related to the interest elements inresponse to the user's inquiry. In this example, the system may send ane-mail or text message to the user's mobile phone with a URL to aparticular webpage by a manufacturer of the product, where that webpageprovides more details regarding the product. The e-mail or text messagemay include additional information to the user, such as the time of dayand location as to when the user may the request, etc.

In some implementations, the targeted content determination module 640determines the format and contents of the targeted content based on allpossible data. For example, the targeted content determination module640 may determine a voice with an appropriate volume, pitch or any otherattribute or no voice at all for the targeted content based on theanalysis result produced by the noise sourcing module 620. In someimplementations, the targeted content determination module 640 analyzesa user's input and other writing or speaking samples to specificallydeduce the user's interaction styles, such as being reserved orauthoritative, and the user data overall to determine the user'sidentity, such as being a female or a teacher, the user's mood, such asbeing bored or motivated, and the user's temperaments, such as beingimpulsive or indecisive.

In some implementations, the targeted content determination module 640examines other user-related data that may reveal user preferences. Asone example, the user's music selections and television viewing patternsmay indicate a preferred tone and speech style with which the user iscomfortable. By analyzing the audio surrounding a user to identify namesof specific songs and television shows perceived by the user, the systemcan retrieve actual sound bites of the songs and the shows as well asrelated metadata from original producers or other curators and analyzethe retrieved data along the desired dimensions.

As another example, the system can, using known APIs, access the user'semail, voicemail, and/or calendar and analyze the data from theemail/voicemail/calendar to determine the user's current mood and mentalstate. For instance, when e-mail/voicemail messages that the user sendstend to be verbose, or when the user's schedule is wide open, the usermay not be rushed and may welcome a descriptive style. When the user'svoicemail messages include heavy breathing or high pitches, or when ameeting reminder has not been acknowledged indicating that the user islate for the meeting, the system may determine that the user is in arush and opt for an instant or delayed delivery depending on the lengthof the content to be delivered.

An example of a suitable data structure for providing targeted contentfor particular users may be as follows:

User ID Language Style Mood/Voice Delivery Rules 987654 English FormalSuccinct Office PC Time-based 876543 French Informal Irreverent MobileGeographic

In this example, users may have a separate profile that indicates theirpreferred language, style of delivery, mood for that delivery, deliverylocation, and rules for that location. In the first example, the userprefers content in English, and presented in a formal style, but in a“rushed” or abbreviated format which is provided to a preferred location(the user's office computer), where the delivery rules are based on time(e.g. office hours). Other styles and moods can represent the type ofresponse, such as human versus robotic, generic versus one associatedwith or from a voice actor (e.g. William Shatner), funky versusauthoritative, upbeat versus stern, and so forth. The delivery can alsobe associated targeted to particular vehicles or locations, such asdelivery to an automotive environment, a train, a boat/ship, anairplane, etc. Each of these vehicles can have an associated widget orapplication to provide the functionality described herein.

As noted above, the targeted content determination module 640 determinesa voice that suits a user's identity, mood and temperament and matchesthe user's interaction style, which may be based on user profiles, andwhich can be useful in any automated communication. But, as describedherein, the system may automatically determine a mood or temperament toapply based on analysis of a user's received utterance.

As one example regarding the use of particular voices for contentprovided to users, an eVite or an electronic greeting card mayincorporate a customized voice in a recording based on the occasion, theintended audience, etc. A child going to a dance party, a woman gettingmarried, and a professor retiring from his university post as well astheir friends and relatives are likely to be receptive to differenttypes of voices, and a customized voice may manifest the intendedmeaning of the sender's message and make a substantial impact on therecipient. In this case, each eVite or greeting card may utilize adifferent voice, and the system or the sender can select an appropriateeVite or greeting card and the associated voice. As another example, aJeopardy-type game may utilize different voices to present differentquestions depending on the contestants, the question categories, theprizes involved, and so on. Doing so may further engage the contestantsand the participating audience and increase the entertainment value ofthe game. In this case, the game may package a variety of voices orsynthesize appropriate voices on the spot based on information providedby users of the game as well as the nature and state of the game.

An example of a suitable data structure for providing a customized voicemay be as follows:

Profile ID Nature Audience Category Difficulty Prize 23 Greeting MotherBirthday N/A N/A card 25 Game Young Sports High $10,000 male

The system could employ a separate profile for each customized voice. Inthe first example (Profile ID 23), the customized voice is for agreeting card intended for a mother's birthday. Therefore, it may be avoice from a relatively older person with tones of respect andsweetness. In the second example (Profile ID 25), the customized voiceis for a game with a young male contestant, where the current categoryis sports, the content is difficult, and the prize is large. As aresult, it may be a male's voice that sounds enthusiastic yet prudent.The customization may be implemented for different levels ofgranularities, taking various factors into consideration. For example,additional information about the audience and the category can beincorporated to fine-tune each customized voice.

The targeted content determination module 640 may then select targetedcontent that matches the user's interaction style and suits the user'sidentity, mood and temperament. As one example, the targeted contentdetermination module 640 may personalize the presentation of a targetedcontent by including a name of a user or a city where the user lives. Asanother example, when a user's inquiry is about the size of a pair ofpants, the targeted content determination module 640 may removechildren's sizes and men's clothes from the targeted content when theuser's age is above a certain threshold and the user is determined to bea woman (where both relative age, and gender, can be determinedautomatically by the system based on the received utterance from theuser). As yet another example, when the user sounds slow andauthoritative and exhibits a keen yet careful attitude, the targetedcontent determination module 640 would converse, with the help of thetext-to-speech module 330, with the user in a similar voice and presentsdetailed contents in an organized fashion to make the user feel engagedand guide the user through the process of learning and ultimatelypurchasing an advertised product. Presenting a customized voice may behelpful for facilitating communication in other, non-advertisementcontexts, as discussed herein.

In addition, the targeted content determination module 640 determineshow to deliver and present the targeted content based on the user data.As one example, the user data may indicate that a user performs a largenumber of activities online and thus may be an experienced internetuser. As noted previously, such user data can be collected from a deviceoperated by the user, including data transmitted as part of a user'sonline activities, such as the user's tweets and Facebook posts, anddata tracking a user's online activities, such as Internet cookies andlogs. The user data may also be collected directly from users throughsurveys or questionnaires. As a result, when the number of onlineactivities performed by the user exceeds a threshold predetermined basedon average user statistics or existing studies, for example, thetargeted content determination module 640 may conclude that the user isan experienced internet user and present the targeted content in aflashy web page to the user.

As another example, since the user may indicate the user's professionand hobbies as well as general preferences and particular items ofinterest in different categories, the user data may reveal that a useris in a very busy profession or otherwise involved in a certain activityat the moment and would prefer a non-intrusive, succinct presentation ofadvertisement. Therefore, when the user data is tagged with certainkeywords, such as “busy”, “occupied”, “focused” and “time-conscious”,for example, the targeted content determination module 640 would chooseto present the targeted content in easy-to-understand text or graphicsvia email.

The targeted content determination module transmits instructions forplaying the targeted content to appropriate hardware and/or softwaremodules or computing devices, which could be a device on which thevoice-controlled advertisement platform 400 is deployed or othersystems. For example, when the voice-controlled advertisement platform400 operates on a mobile device, the targeted content determinationmodule 640 may instruct the mobile device to launch a web browser todisplay the targeted content in a particular web page. In someimplementations, the targeted content determination module 640 utilizesapplication programming interfaces (APIs) existing on the operatingsystem of a client device (e.g., a user's mobile device) andfunctionality embodied in the operating system and other applications ofthe client device to automatically present targeted content to a user.

The targeted content determination module 640 saves the targeted contentalong with the inquiries to which they respond in the targeted contentdatabase 490. As discussed above, targeted content previously presentedto a user could be used to help determine the subject matter of theuser's future inquiries. Accordingly, when the voice-controlledadvertisement platform is deployed on both a user device and a servercomputer, the targeted content determination module 640 may also predictadditional content that may be of interest to the user and preload themto the user device to reduce the response time to the user. As issimilar to the preloading of custom grammars or language models, thepreloading of content may be performed periodically or at specifictimes. It may be performed when the load of the network between theserver computer and the client device is relatively low, when additionalcontent of interest become available, and so on. The preloaded contentmay then be stored in a local storage or in a cache on the client deviceto further expedite the utilization of the content.

In some implementations, the targeted content determination module 640may classify inquiries from users and aggregate the targeted contentpresented in response to each class of inquiries. In someimplementations, it may define user groups based on various attributes,such as age, location, profession, etc., and aggregate the targetedcontents presented to each user group. The user groups may be defined byallowing a certain range for each of the attributes. For example, allthe users of ages between eighteen and twenty-one or living within afive-mile radius may belong to the same user group. It may thendetermine which contents are most frequently delivered with respect to aclass of inquiries and/or a user group and preload or cache those andrelated advertisements on select user devices to reduce the responsetime to any member in the particular user group. The most frequentlydelivered contents may be determined by considering the deliveryfrequencies within a certain timeframe above a certain percentile. Itmay also be determined by limiting the total volume of preloadedcontents based on the capacity of the network connection between theserver computer and the client device, the size of the cache on theclient device, the number of inquiries received from the user within acertain timeframe, and so on. The contents may also be furtherclassified into various categories, by the nature of the contents, thesize of the contents, the manner of delivery, and so on. Thedetermination of the most frequently delivered contents may be limitedto contents in specific categories; otherwise, the preloading ofcontents in different categories may be performed according to differentschedules.

As one example, user inquiries with respect to a specific scene of amovie may tend to be directed to the song playing in the background, andwhen users inquire about the song, they often are also interested inknowing more about the artist performing the song. Therefore, havingdelivered a large amount of content regarding the movie and the song tocertain members of the user group corresponding to the intended audienceof the movie or of the song, the targeted content determination module640 may preload or cache advertisements of the artist, the movie or thesong on the devices of other members of the user group. As anotherexample, a video game series might have become very popular among aparticular age group, and thus advertisements of any video game in theseries have been extensively presented to users of that group. Thetargeted content determination module 640 may therefore preload or cacheadvertisements related to the video game series for all users of thatparticular age group on select user devices. It may also preload orcache related advertisements, such as those of related video game seriesand those of other items that are of common to that age group. As moreadvertisements are being delivered, the targeted content determinationmodule 640 may iteratively re-determine which advertisements have beenmostly accessed and push appropriate advertisements to select userdevices according to a predetermined schedule.

FIG. 9 is a flowchart illustrating an example process of preloadingcommon responses. At block 905, the targeted content determinationmodule 640 saves all the targeted contents presented to users by usergroup, content category, inquiry type, or any other classification groupin the targeted content database 480. Each piece of content, such as anadvertisement, is associated with metadata for analysis purposes. Anexample of a generic data structure that can be used for a targetedadvertisement can be as follows:

ID Category Nature Inquiry User 10 Cars Aston Martin Inquiry 1 User 1 20Food Soufflé Inquiry 20 User 20

In the first example (ID 10), the targeted advertisement was previouslydelivered to a device of User 1, where information regarding the usercan be found by following the link to that user's record in the userdatabase 450. Furthermore, it was delivered in response to Inquiry 1,where information regarding the inquiry can be found by following thelink to that inquiry's record also in the targeted content database 480.In addition, the targeted advertisement is classified into the carscategory, as it is about an Aston Martin model manufactured in a certainyear. Similarly, in the second example (ID 20), the targetedadvertisement was previously delivered to a device of User 20 inresponse to Inquiry 20. It is classified into the food category, as itis about a certain type of soufflé made by a certain restaurant. For aparticular user group, for example, the targeted content determinationmodule 640 may identify the targeted advertisements from the informationin the user field.

For a select classification group, at block 910, the targeted contentdetermination module 640 identifies the advertisements that have beenmost commonly presented. Specifically, it keeps track of every deliverywithin a certain timeframe for each targeted advertisement. It may thenrank the targeted advertisements in terms of delivery frequencies ordetermine whether each delivery frequency is sufficiently high based ona certain threshold. It may re-identify the advertisements that havebeen most commonly presented periodically, when a certain amount oftargeted advertisements have been added to the database, and so on. Thetargeted content determination module 640 also identifies relatedadvertisements on the same or similar subject matters. For example, ifthe targeted advertisement in the first example is selected forpreloading, the targeted content determination module 640 may alsoselect contents from the same car manufacturer, manufacturers of similarcars, manufactures of car products, such as windshield fluids,advertisers of other items that might be of interest to the same usergroup, and so on.

At block 915, the targeted content determination module 640 preloads theidentified advertisements to appropriate devices. For example, if theselect classification group is the group of users between ages 18 and25, the appropriate devices would be the devices of these users on whichthe voice-controlled advertisement platform is deployed. As noted above,the preloading can be performed periodically or at select times. It maypreload all the relevant content at once or spread the preloading overseveral scheduled times. The preloading is likely to reduce the time ittakes to respond to the inquiries from the classification group overall.The preloaded data can be stored in a cache on each device to furtherreduce the response time.

At step 920, in response to a relevant inquiry received by a clientdevice of a member of the select classification group, the client devicemay quickly present some of the preloaded advertisements to the user.The preloading may include, in addition to the targeted advertisements,the inquiries which have triggered the presentation of the targetedadvertisements. In that case, when a new inquiry is one of thosepreloaded inquiries, the client device may immediately present thecorresponding targeted advertisement. In the absence of the preloadedinquiries, the client device would wait for an instruction from theserver computer of presenting a preloaded advertisement, instead of anactual advertisement when no preloading has occurred.

CONCLUSION

Those skilled in the art will appreciate that the actual implementationof a data storage area may take a variety of forms, and the phrase “datastorage area” is used herein in the generic sense to refer to any areathat allows data to be stored in a structured and accessible fashionusing such applications or constructs as databases, tables, linkedlists, arrays, and so on. Those skilled in the art will furtherappreciate that the depicted flow charts may be altered in a variety ofways. For example, the order of the blocks may be rearranged, blocks maybe performed in parallel, blocks may be omitted, or other blocks may beincluded.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” As used herein, the terms “connected,”“coupled,” or any variant thereof means any connection or coupling,either direct or indirect, between two or more content elements; thecoupling or connection between the content elements can be physical,logical, or a combination thereof. Additionally, the words “herein,”“above,” “below,” and words of similar import, when used in thisapplication, refer to this application as a whole and not to anyparticular portions of this application. Where the context permits,words in the above Detailed Description using the singular or pluralnumber may also include the plural or singular number respectively. Theword “or,” in reference to a list of two or more items, covers all ofthe following interpretations of the word: any of the items in the list,all of the items in the list, and any combination of the items in thelist.

The above Detailed Description of examples of the invention is notintended to be exhaustive or to limit the invention to the precise formdisclosed above. While specific examples for the invention are describedabove for illustrative purposes, various equivalent modifications arepossible within the scope of the invention, as those skilled in therelevant art will recognize. For example, while processes or blocks arepresented in a given order, alternative implementations may performroutines having steps, or employ systems having blocks, in a differentorder, and some processes or blocks may be deleted, moved, added,subdivided, combined, and/or modified to provide alternative orsubcombinations. Each of these processes or blocks may be implemented ina variety of different ways. Also, while processes or blocks are attimes shown as being performed in series, these processes or blocks mayinstead be performed or implemented in parallel, or may be performed atdifferent times. Further any specific numbers noted herein are onlyexamples: alternative implementations may employ differing values orranges.

The teachings of the invention provided herein can be applied to othersystems, not necessarily the system described above. The contentelements and acts of the various examples described above can becombined to provide further implementations of the invention. Somealternative implementations of the invention may include not onlyadditional elements to those implementations noted above, but also mayinclude fewer elements. The assignee's U.S. patent application Ser. No.13/597,017, filed Aug. 28, 2012, and Ser. No. 13/749,798, filed Jan. 25,2013, are incorporated herein by reference. Aspects of the invention canbe modified, if necessary, to employ the systems, functions, andconcepts of the various references described above to provide yetfurther implementations of the invention.

These and other changes can be made to the invention in light of theabove Detailed Description. While the above description describescertain examples of the invention, and describes the best modecontemplated, no matter how detailed the above appears in text, theinvention can be practiced in many ways. Details of the system may varyconsiderably in its specific implementation, while still beingencompassed by the invention disclosed herein. As noted above,particular terminology used when describing certain features or aspectsof the invention should not be taken to imply that the terminology isbeing redefined herein to be restricted to any specific characteristics,features, or aspects of the invention with which that terminology isassociated. In general, the terms used in the following claims shouldnot be construed to limit the invention to the specific examplesdisclosed in the specification, unless the above Detailed Descriptionsection explicitly defines such terms. Accordingly, the actual scope ofthe invention encompasses not only the disclosed examples, but also allequivalent ways of practicing or implementing the invention under theclaims.

To reduce the number of claims, certain aspects of the invention arepresented below in certain claim forms, but the applicant contemplatesthe various aspects of the invention in any number of claim forms. Forexample, while only one aspect of the invention is recited as ameans-plus-function claim under 35 U.S.C. sec. 112, sixth paragraph,other aspects may likewise be embodied as a means-plus-function claim,or in other forms, such as being embodied in a computer-readable medium.(Any claims intended to be treated under 35 U.S.C. § 112, ¶6 will beginwith the words “means for”, but use of the term “for” in any othercontext is not intended to invoke treatment under 35 U.S.C. § 112, ¶6.)Accordingly, the applicant reserves the right to pursue additionalclaims after filing this application to pursue such additional claimforms, in either this application or in a continuing application.

The invention claimed is:
 1. A method of processing audio information,comprising: receiving a user's voice inquiry from an electronic device;determining at least one interest element related to the voice inquiryfrom: media content currently provided to the user by the electronicdevice, media content from the user's environment, wherein the mediacontent from the user's environment includes background sounds presentin the user's environment, and media content previously provided to theuser as responses to previous voice inquiries by the user; andgenerating a response related to the determined interest element to theuser's voice inquiry.
 2. The method of claim 1, wherein the electronicdevice is selected from one of a television, a radio, a gaming console,a laptop, a tablet, a convertible laptop, a cellular phone, a wearabledevice, and a kiosk, and wherein presenting the response related to thedetermined interest element includes presenting the response via theelectronic device.
 3. The method of claim 1, wherein the media contentincludes media content from at least one of a movie, a televisionprogram, a radio show, a song, a video, a game, and an advertisement. 4.The method of claim 1, further comprising determining the media contentrepresenting the user's environment based on information related to theuser's location.
 5. The method of claim 1, further comprising: obtainingpersonal data related to the user; and determining the at least oneinterest element based, at least in part, on the obtained personal data.6. The method of claim 1, wherein the media content from the user'senvironment further includes scenes.
 7. An audio processing system,comprising one or more processors configured to: receive a user's voiceinquiry from an electronic device; determine at least one interestelement related to the voice inquiry from: media content currentlyprovided to the user by the electronic device, media content from theuser's environment, wherein the media content from the user'senvironment includes background sounds present in the user'senvironment, and media content previously provided to the user asresponses to previous voice inquiries by the user; and generate aresponse related to the determined interest element to the user's voiceinquiry.
 8. The system of claim 7, wherein: the at least one electronicdevice is selected from one of a television, a radio, a gaming console,a laptop, a tablet, a convertible laptop, a cellular phone, a wearabledevice, a dashboard, and a kiosk, and the one or more processors isconfigured to present the response on the device.
 9. The system of claim7, wherein the media content includes at least one of media content froma movie, a television program, a radio show, a song, a video, a game,and an advertisement.
 10. The system of claim 7, wherein the one or moreprocessors is further configured to determine the media content from theuser's environment based on information on a current time or a currentlocation of the user.
 11. The system of claim 7, wherein the one or moreprocessors is further configured to: obtain personal data related to theuser; and determine the at least one interest element based, at least inpart, on the personal data.
 12. The system of claim 11, wherein theobtained personal data indicates the user's identity, habits,preferences, and/or moods.
 13. The system of claim 11, wherein the oneor more processors is configured to present the response based on theobtained personal data.
 14. The system of claim 7, wherein the one ormore processors is further configured to output the response via theelectronic device.
 15. The system of claim 14, wherein when the at leastone interest element is determined from media content currently providedto the user on the electronic device, the one or more processors isfurther configured to: output the response in association with the mediacontent currently provided to the user on the electronic device.
 16. Themethod of claim 5, wherein the personal data indicates at least one ofthe user's identity and preferences.
 17. The method of claim 1, furthercomprising outputting the response via the electronic device.
 18. Themethod of claim 17, wherein when the at least one interest element isdetermined from media content currently provided to the user on theelectronic device, the method further comprising: outputting theresponse in association with the media content currently provided to theuser on the electronic device.